ios webRTC实现屏幕共享功能
杂谈:
使用webRTC开发音视频通话,与会议功能,已经有一年时间,一年时间从技术调研,到方案实施,我所起到的作用不是很大,技术方向主导依然是技术总监,在这里也对总监的技术表示钦佩,佩服他的敬业精神,以及对技术的追求,在音视频编解码这一块也是很有话语权的,40多岁的人,每天依然可以在公司工作到9点下班,这种精神头,年轻人也不一定有.
通过对webRTC在ios端的实现与在web端的实现(我负责公司ios端功能开发并兼职负责webRTC, web端的开发功能),收获了很多,对比两端实现,大致基本相同,在实现[屏幕共享功能上,这一点差距和思想上有很大的区别也有很大的相同点.接下来我想先讲iOS端的实现,在下一篇文章中将web端的实现,避免混淆,两篇做一个对比比较清晰
前言:
实现ios端屏幕共享功能,随着直播的兴起,录播显得尤为重要,ios9以后苹果终于开放了replaykit框架,用于实现应用内的录制,可以实现应用内屏幕直播,但是不能实现系统的录制功能,在ios12后苹果终于推出了replaykit2实现系统屏幕录制的功能,但是要实现系统内的录制功能,就需要创建extensionApp在另一个进程中录制屏幕数据,在主app与扩展app通信的过程中就涉及到进程间的通信,如果在项目中应用到三方或者逻辑组封装好的编码接口,可以直接调用传数据进行编码,实现将录制数据传到远端,如果需要将录制数据回传到主app进行编解码,那就需要涉及到进程间的通信,这就是这篇文章的主要内容
主要解决问题:
实现录制功能的流程:
1.创建工程,在编辑器下方点击+
出现如下图所示点击图中选中的内容下一步填写名称(随便写,更具项目命名规则来就行),其他内容都是用默认设置就可以
Finish后项目中多了几个文件,我们关心的只是名为 SampleHandler.h .m 的文件
2.进入到Project 看到extensionApp的配置项,这里有一个知识点就是
主app Bundle ID为: HeLi.LTD.screen.com
extension Bundle ID : HeLi.LTD.screen.com.upload
extension UI Bundle ID : HeLi.LTD.screen.com.setUpUI
在创建证书的时候需要注意这一点以上内容便是创建extensionApp的内容,
2.解决进程间的通信问题
进程间的通信ios下有多种,socket,剪切板,共享内存, 一共有9种好像,具体的没有过多研究,这里主要讲socket与进程间的通知(notificationcenter)实现通信功能,
在主app中启动socket服务端(server),在extension端实现客户端(client),服务端在应用调用视频时便启动socket连接,待客户端回传数据(并不一定会启动屏幕共享,可以优化)
在extensionApp中开始共享时创建socket连接
- (void)setupSocket
{
if (self.socket.isConnected) {
return;
}
self.sockets = [NSMutableArray array];
self.recvBuffer = (NTESTPCircularBuffer *)malloc(sizeof(NTESTPCircularBuffer)); // 需要释放
NTESTPCircularBufferInit(self.recvBuffer, kRecvBufferMaxSize);
// self.queue = dispatch_queue_create("com.netease.edu.rp.server", DISPATCH_QUEUE_SERIAL);
self.queue = dispatch_get_main_queue();
self.socket = [[GCDAsyncSocket alloc] initWithDelegate:self delegateQueue:self.queue];
self.socket.IPv6Enabled = NO;
NSError *error;
// [self.socket acceptOnUrl:[NSURL fileURLWithPath:serverURL] error:&error];
[self.socket acceptOnPort:8999 error:&error];
[self.socket readDataWithTimeout:-1 tag:0];
if (error == nil)
{
NSLog(@"开启成功");
// [[NSRunLoop mainRunLoop]run];//目的让服务器不停止
[self setTimer];
}
else
{
NSLog(@"开启失败");
[self.socket disconnect];
[self setupSocket];
}
NSNotificationCenter *center =[NSNotificationCenter defaultCenter];
[center addObserver:self
selector:@selector(defaultsChanged:)
name:NSUserDefaultsDidChangeNotification
object:nil];
}客户端的建立
- (void)broadcastStartedWithSetupInfo:(NSDictionary *)setupInfo {
if (!self.connected) {
[self.socket disconnect];
}
if (!self.socket.isConnected) {
[self setupSocket];
}
}
- (void)setupSocket
{
_recvBuffer = (NTESTPCircularBuffer *)malloc(sizeof(NTESTPCircularBuffer)); // 需要释放
NTESTPCircularBufferInit(_recvBuffer, kRecvBufferMaxSize);
self.queue = dispatch_queue_create("com.netease.edu.rp.client", DISPATCH_QUEUE_SERIAL);
self.socket = [[GCDAsyncSocket alloc] initWithDelegate:self delegateQueue:self.queue];
// self.socket.IPv6Enabled = NO;
// [self.socket connectToUrl:[NSURL fileURLWithPath:serverURL] withTimeout:5 error:nil];
NSError *error;
[self.socket connectToHost:_ip onPort:8999 error:&error];
[self.socket readDataWithTimeout:-1 tag:0];
NSLog(@"setupSocket:%@",error);
if (error == nil)
{
NSLog(@"====开启成功");
}
else
{
NSLog(@"=====开启失败");
}
} 发送数据 在获取到录制的数据流后将流编码后发送
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
switch (sampleBufferType) {
case RPSampleBufferTypeVideo:
{
if (!self.connected)
{
return;
}
// if(self.connected){
if(_finish){
NSError *error = [NSError errorWithDomain:NSStringFromClass(self.class)
code:0
userInfo:@{
NSLocalizedFailureReasonErrorKey:@"屏幕共享已结束"
}];
[self finishBroadcastWithError:error];
}
if( CMSampleBufferDataIsReady(sampleBuffer) || CMSampleBufferIsValid(sampleBuffer) || CMSampleBufferGetNumSamples(sampleBuffer)){
[self sendVideoBufferToHostApp:sampleBuffer];
}
}
break;
case RPSampleBufferTypeAudioApp:
// Handle audio sample buffer for app audio
break;
case RPSampleBufferTypeAudioMic:
// Handle audio sample buffer for mic audio
break;
default:
break;
}
}
- (void)sendVideoBufferToHostApp:(CMSampleBufferRef)sampleBuffer {
if (!self.socket)
{
return;
}
CFRetain(sampleBuffer);
long curMem = [self getCurrentMemory];
if ((self.eventMemory > 0
&& ((curMem - self.eventMemory) > 5))
|| curMem > 35) {
//当前内存暴增5M以上，或者总共超过45M，则不处理
CFRelease(sampleBuffer);
return;
};
dispatch_async(self.videoQueue, ^{ // queue optimal
@autoreleasepool {
if (self.frameCount > 0)
{
CFRelease(sampleBuffer);
return;
}
self.frameCount ++ ;
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CFStringRef RPVideoSampleOrientationKeyRef = (__bridge CFStringRef)RPVideoSampleOrientationKey;
NSNumber *orientation = (NSNumber *)CMGetAttachment(sampleBuffer, RPVideoSampleOrientationKeyRef,NULL);
switch ([orientation integerValue]) {
case 1:
self.orientation = NTESVideoPackOrientationPortrait;
break;
case 6:
self.orientation = NTESVideoPackOrientationLandscapeRight;
break;
case 8:
self.orientation = NTESVideoPackOrientationLandscapeLeft;
break;
default:
break;
}
// To data
NTESI420Frame *videoFrame = nil;
videoFrame = [NTESYUVConverter pixelBufferToI420:pixelBuffer
withCrop:self.cropRate
targetSize:self.targetSize
andOrientation:self.orientation];
CFRelease(sampleBuffer);
// To Host App
if (videoFrame){
NSData *raw = [videoFrame bytes];
//NSData *data = [NTESSocketPacket packetWithBuffer:raw];
NSData *headerData = [NTESSocketPacket packetWithBuffer:raw];
if (!_enterBack) {
if (self.connected) {
[self.socket writeData:headerData withTimeout:-1 tag:0];
[self.socket writeData:raw withTimeout:-1 tag:0];
}
}
}
self.frameCount --;
};
});
self.eventMemory = [self getCurrentMemory];
}主要代码就是上面的实现过程,具体的代码,我会放在demo中
2.CFNotificationCenterRef 进程的通知中心
在SampleHandler.m中重写init方法订阅通知中心
- (instancetype)init {
if(self = [super init]) {
_targetSize = CGSizeMake(414, 812);
_cropRate = 15;
_orientation = NTESVideoPackOrientationPortrait;
_ip = @"127.0.0.1";
_serverPort = @"8999";
_clientPort = [NSString stringWithFormat:@"%d", arc4random()%9999];
_videoQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
CFStringRef name = CFSTR("customName");
CFNotificationCenterRef center = CFNotificationCenterGetDarwinNotifyCenter();
CFNotificationCenterAddObserver(center,
(const void *)self,
Callback,
name,
NULL,
kCFNotificationDeliverImmediately);
CFStringRef finishName = CFSTR("StopScreen");
CFNotificationCenterRef finishCenter = CFNotificationCenterGetDarwinNotifyCenter();
CFNotificationCenterAddObserver(finishCenter,
(const void *)self,
FinishCallback,
finishName,
NULL,
kCFNotificationDeliverImmediately);
}
return self;
}
BOOL _enterBack = false;
static void Callback(CFNotificationCenterRef center,
void *observer,
CFStringRef name,
const void *object,
CFDictionaryRef userInfo)
{
_enterBack = true;
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(3 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
_enterBack = false;
});
}
BOOL _finish = false;
static void FinishCallback(CFNotificationCenterRef center,
void *observer,
CFStringRef name,
const void *object,
CFDictionaryRef userInfo)
{
_finish = true;
}在发送端(主app,),发送通知消息
CFStringRef finishName = CFSTR("StopShare");
CFNotificationCenterRef finishCenter = CFNotificationCenterGetDarwinNotifyCenter();
CFNotificationCenterAddObserver(finishCenter,
(const void *)self,
FinishCallback,
finishName,
NULL,
kCFNotificationDeliverImmediately);
BOOL _finish = false;
static void FinishCallback(CFNotificationCenterRef center,
void *observer,
CFStringRef name,
const void *object,
CFDictionaryRef userInfo)
{
_finish = true;
}
3.解决替换webRTC视频源
在主app中接收到socket发送来的消息后,将数据转为CMSampleBufferRef,GCDAsySocket只能发送NSData的数据,所以需要一个转换的过程,将yuv数据转为nsdata数据在转回CMSampleBufferRef
- (void)onRecvData:(NSData *)data
{
dispatch_async(dispatch_get_main_queue(), ^{
//将数据zhuanweiI420,yuv数据 4:2:0
NTESI420Frame *frame = [NTESI420Frame initWithData:data];
CMSampleBufferRef sampleBuffer = [frame convertToSampleBuffer];
if (sampleBuffer == NULL) {
return;
}
if(!_isStart){
if (_startScreenBlock) {
_startScreenBlock(true);
}
// if(!_StartScreen){
//
// }else{
//
// if (_startScreenBlock) {
// _startScreenBlock(false);
// }
// }
_isStart = true;
_Running = false;
[_capture stopCapture]; //收到数据代表录制打开,需要暂停本地摄像头的数据
}else{
}
if([UIScreen mainScreen].isCaptured){
if(_Running){
_isStart = true;
_Running = false;
[_capture stopCapture];
if (_startScreenBlock) {
_startScreenBlock(true);
}
}
}
// if (self.StartScreen) {
int64_t timeStampNs =
CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * NSEC_PER_SEC; //NSEC_PER_SEC 这个很关键,
CVPixelBufferRef rtcPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
RTCCVPixelBuffer *cpvX = [[RTCCVPixelBuffer alloc]initWithPixelBuffer:rtcPixelBuffer];
RTCVideoFrame *aframe = [[RTCVideoFrame alloc]initWithBuffer:cpvX rotation:self.rotation timeStampNs:timeStampNs];
//替换数据源
[_videoSource capturer:_capture didCaptureVideoFrame:aframe];
// }
CFRelease(sampleBuffer);
});
}5.解决录制进程50M内存限制
由于系统的限制extensionApp只要50M的运行空间,超过就会被系统杀死,在CMSampleBufferRef提取yuv数据转I420数据的过程中,耗时较多,在数据刷新快的时候会造成内存开销变大,导致录制进程崩溃问题,这里补充一个知识点
录制的数据是根据屏幕的刷新率来确定截取的帧数,屏幕没有变化,则不会有视频流被捕获,只有屏幕内容变化的时候才会得到录制的数据,屏幕刷新快相应的得到更多的数据,造成瞬时需要处理更多的数据,所以解决内存限制的思路是减少处理数据的量,从而限制内存的占用,
首先我们通过监控内存的数据变化,来确定是否接收处理录制的视频数据
//获取当前进程所占的内存
- (double)getCurrentMemory
{
task_basic_info_data_t taskInfo;
mach_msg_type_number_t infoCount = TASK_BASIC_INFO_COUNT;
kern_return_t kernReturn = task_info(mach_task_self(),
TASK_BASIC_INFO,
(task_info_t)&taskInfo,
&infoCount);
if (kernReturn != KERN_SUCCESS
) {
return NSNotFound;
}
NSLog(@"%f",taskInfo.resident_size / 1024.0 / 1024.0);
return taskInfo.resident_size / 1024.0 / 1024.0;
}
//在次方法中调用
- (void)sendVideoBufferToHostApp:(CMSampleBufferRef)sampleBuffer {
long curMem = [self getCurrentMemory];
CFRetain(sampleBuffer);
if ((self.eventMemory > 0
&& ((curMem - self.eventMemory) > 5))
|| curMem > 35) {
//当前内存暴增5M以上，或者总共超过35M，则不处理
//这里写35M是为本次计算后的数据处理留出空间,超过将不再处理数据,直接释放流信息
CFRelease(sampleBuffer);//释放数据
return;
};
//此处处理数据的编码转换
}